A Pitfall in Determining the Optimal Feature Subset Size

Feature selection researchers often encounter a peaking phenomenon: a feature subset can be found that is smaller but still enables building a more accurate classifier than the full set of all the candidate features. However, the present study shows that this peak may often be just an artifact due to the still too common mistake in pattern recognition — that of not using an independent test set.